Provisioning with Terraform

Terraform, in terraform/hetzner-main/, creates the VMs on the Proxmox host and sets their hardware, network and cloud-init settings. It doesn't install software inside them; Ansible does that (see Configuring with Ansible).

Only the hetzner-main VMs are in Terraform. The Proxmox host, Amergin, Secops, Vaultwarden and Puca were set up by hand (or through Hetzner's consoles) and are configured by Ansible only.

The VMs

Each VM has its own file:

File	Resource name	VM	VMID	IP	CPU	RAM	Disk
docker-build.tf	ansible_server	docker-build	112	10.0.0.12	2	8 GB	50 GB
webserver.tf	webserver_test	webserver	114	10.0.0.14	4	4 GB	16 GB
api-gateway.tf	api_gateway	api-gateway	117	10.0.0.17	2	4 GB	16 GB
services-user.tf	services_user	services-user	118	10.0.0.18	4	8 GB	128 GB
services-staff.tf	services_staff	services-staff	119	10.0.0.19	4	8 GB	64 GB
synthesis-legacy.tf	synthesis_legacy	synthesis-legacy	121	10.0.0.21	4	16 GB	32 GB

Some resource names are left over from earlier roles (ansible_server, webserver_test). They're kept because renaming a resource makes Terraform want to destroy and recreate the VM.

Every VM:

• is a full clone of debian-13-template (VMID 9000) on the node proxmox, stored on local;
• has one network card on vmbr1, a static IP, gateway 10.0.0.254, and DNS 1.1.1.1 / 1.0.0.1;
• uses cloud-init to create the user ansible-control with the admins' SSH keys;
• has the QEMU guest agent enabled.

The template is built by the Proxmox host playbook's template tag. See Proxmox Host.

How Terraform reaches Proxmox

The provider is telmate/proxmox (version 3.0.2-rc07, configured in docker-build.tf). It uses the API token terraform@pam!tf-token.

The Proxmox API listens on 10.0.0.254:8006, which you can only reach from inside the VM network. terraform/hetzner-main/tf.sh handles this for you. It:

1. fetches the token secret from Vaultwarden (Proxmox Terraform Token);
2. opens an SSH tunnel: ssh -f -N -L 8006:10.0.0.254:8006 $PROXMOX_JUMP_HOST;
3. runs Terraform against https://localhost:8006;
4. closes the tunnel when Terraform exits.

It deliberately doesn't go through proxmox.abair.ie. That name is served by the webserver VM, so a stopped webserver would break every Terraform run. On a fresh install, it would also make the webserver a requirement for creating the webserver.

make provision PROXMOX_JUMP_HOST=proxmox-host

make provision runs tf.sh apply -auto-approve. It creates missing VMs and applies changes to existing ones without asking for confirmation. make wait runs provision first and then sleeps 30 seconds. make setup-all depends on wait, so it also runs a full terraform apply.

State

Terraform state is stored locally, in terraform/hetzner-main/terraform.tfstate on the admin's machine. It isn't committed (*.tfstate is in .gitignore) and isn't shared. Whoever runs make provision needs the current state file.

Before you run make provision

Two things to check first:

1. Stale resource in state. The current state still contains logging_server (the old observability VM, VMID 115), which has no .tf file any more. A terraform apply will try to destroy VMID 115, or drop it from state if it's already gone. If that VM must survive, run terraform state rm proxmox_vm_qemu.logging_server first.
2. Terraform version. The tooling container runs Terraform 1.9.8, but the state file was last written by 1.15.8. Terraform refuses to read state written by a newer version. Either raise TERRAFORM_VERSION in docker/Dockerfile, or plan with the newer version before applying.

Run a plan first (./tf.sh plan inside make docker-shell) and read it before applying.

Provisioning is separate from configuration

The per-VM setup-* and update-* targets only run Ansible against a VM that already exists. They never run Terraform. Create or resize a VM with make provision, then configure it.

Changing a VM's size

The VMs have no CPU or memory hotplug, so a CPU or memory change only takes effect after a full stop and start. Make the change on the Proxmox host, then copy the new values into the .tf file so Terraform sees no difference:

qm shutdown <vmid>
qm set <vmid> --cores <N> --memory <MB>
qm start <vmid>

Changing the VM config regenerates its cloud-init drive, which gives the VM new SSH host keys. Afterwards, clear the old ones with ssh-keygen -R <ip> and ssh-keygen -R <alias>.

A disk can grow while the VM keeps running: qm resize <vmid> scsi0 +<N>G on the host, then growpart and resize2fs inside the VM. This is how services-user went from 32 GB to 128 GB.

This matters most for the webserver and docker-build. Restarting the webserver takes every public site offline. Restarting docker-build takes down the registry every other VM pulls from. Always use the procedure above for those two, never terraform apply.

Adding a VM

See Making Changes → Adding a VM.